{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "513b9069-6aac-40cd-b232-0a2e87192375",
   "metadata": {},
   "source": [
    "### 删除无用数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6e23f6-93bb-4127-984e-21a6d194ecf4",
   "metadata": {},
   "source": [
    "下载的原版lfw和lfwc都是包含全部人的，可以用来预训练模型  \n",
    "如果要进行针对311个人识别进行的训练需要把多余的数据去掉"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0693810-e20f-4dd1-84d2-5217f3341a02",
   "metadata": {},
   "source": [
    "**根据lfw官网 图片集中有几处错误，在这里将本次比赛涉及的错误进行修正**  \n",
    "- train:  \n",
    "Recep_Tayyip_Erdogan_0004 is incorrect (it is an image of Abdullah Gul)  M  \n",
    "Michael_Schumacher_0008 is incorrect (it is an image of Rubens Barrichello)  M\n",
    "\n",
    "- test:  \n",
    "Mahmoud_Abbas_0012 is incorrect (it is an image of Hamad Bin Isa al-Khalifa)  D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d41c1078-b961-48c1-b1cc-d89257885a5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import glob\n",
    "\n",
    "# 获取三个数据集的文件\n",
    "lfw_list = glob.glob('./lfw/*')  # 原版lfw \n",
    "train_list = glob.glob('./dataset/train/*')  # 训练集\n",
    "lfwc_list = glob.glob('./lfwc/*')  # 经过校对的lfw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8830bca4-77a4-44a6-8a70-61e4809c0da0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lfw_list len: 311, sample: ./lfw_processed/Gordon_Brown\n",
      "train_list len: 311, sample: ./dataset/train/JK_Rowling\n",
      "lfwc_list len: 311, sample: ./lfwc_processed/Billy_Crystal\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print(f\"lfw_list len: {len(lfw_list)}, sample: {lfw_list[1]}\")\n",
    "print(f\"train_list len: {len(train_list)}, sample: {train_list[2]}\")\n",
    "print(f\"lfwc_list len: {len(lfwc_list)}, sample: {lfwc_list[1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "79f3e714-c369-4bec-bae8-a6e07a760777",
   "metadata": {},
   "outputs": [],
   "source": [
    "lfw_names = []\n",
    "train_names = []\n",
    "lfwc_names = []\n",
    "\n",
    "# 获取三个数据集的文件列表\n",
    "for item in lfw_list:\n",
    "    lfw_names.append(item.split('/')[-1])\n",
    "    \n",
    "for item in train_list:\n",
    "    train_names.append(item.split('/')[-1])\n",
    "    \n",
    "for item in lfwc_list:\n",
    "    lfwc_names.append(item.split('/')[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a472bfa-64e9-4be0-8263-3bc272c65dbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lfw_names len: 5749, sample: Kevin_Garnett\n",
      "train_names len: 311, sample: JK_Rowling\n",
      "lfwc_names len: 5750, sample: Ahmed_Ahmed\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print(f\"lfw_names len: {len(lfw_names)}, sample: {lfw_names[1]}\")\n",
    "print(f\"train_names len: {len(train_names)}, sample: {train_names[2]}\")\n",
    "print(f\"lfwc_names len: {len(lfwc_names)}, sample: {lfwc_names[1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "91ce2509-d20e-4a81-8ce6-339c39e9fb59",
   "metadata": {},
   "outputs": [],
   "source": [
    "lfw_delete_list = []\n",
    "lfwc_delete_list = []\n",
    "\n",
    "# 生成需要删除的文件列表\n",
    "for item in lfw_names:\n",
    "    if item not in train_names:\n",
    "        lfw_delete_list.append('./lfw_processed/'+item+'/')\n",
    "        lfwc_delete_list.append('./lfwc_processed/'+item+'/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9e708dfd-156c-441b-87ee-dce0f737b229",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5438\n",
      "5438\n",
      "./lfw_processed/Christina_Sawaya/\n",
      "./lfwc_processed/Christina_Sawaya/\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "print(len(lfw_delete_list))  # 5749 - 311 = 5438\n",
    "print(len(lfwc_delete_list))\n",
    "print(lfw_delete_list[0])\n",
    "print(lfwc_delete_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e5575a2e-ea55-41ad-8c3b-c783c8e0f8c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删除多余数据\n",
    "for item in lfw_delete_list:\n",
    "    os.system('rm -r'+' '+item)\n",
    "\n",
    "for item in lfwc_delete_list:\n",
    "    os.system('rm -r'+' '+item)\n",
    "\n",
    "new_lfw_list = glob.glob('./lfw_processed/*')\n",
    "new_lfw_names = []\n",
    "for item in new_lfw_list:\n",
    "    new_lfw_names.append(item.split('/')[-1])\n",
    "    \n",
    "new_lfwc_list = glob.glob('./lfw_processed/*')\n",
    "new_lfwc_names = []\n",
    "for item in new_lfwc_list:\n",
    "    new_lfwc_names.append(item.split('/')[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5923dcef-0b0f-4675-973b-60a84ff4f4d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# check \n",
    "# 这里出现Flase也没有问题  到文件夹里检查一下就可以\n",
    "print(new_lfw_names==train_names)\n",
    "print(new_lfwc_names==train_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f756160a-6c32-4a17-b0e5-330fd405a76d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "311"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(new_lfw_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "41941279-a567-4be8-ae34-bfd0f923babd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "311"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(new_lfwc_names)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
